Similarity-Based Non-Scorable Response Detection for Automated Speech Scoring

This study provides a method that identifies problematic responses which make automated speech scoring difficult. When automated scoring is used in the context of a high stakes language proficiency assessment, for which the scores are used to make consequential decisions, some test takers may have an incentive to try to game the system in order to artificially inflate their scores. Since many automated proficiency scoring systems use fluency features such as speaking rate as one of the important features, students may engage in strategies designed to manipulate their speaking rate as measured by the system. In order to address this issue, we developed a method which filters out nonscorable responses based on text similarity measures. Given a test response, the method generated a set of features which calculated the topic similarity with the prompt question or the sample responses including relevant content. Next, an automated filter which identified these problematic responses was implemented using the similarity features. This filter improved the performance of the baseline filter in identifying responses with topic problems.